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Two kinds of efforts at instruction^, iaprovetent in 
higher education, student evaluation of teachers ana;faculty 
development programs on caapusesv f all jshort of rfeactoing their 
theoretical goals. 'Student evaluations are neither reliable aor valid 
measures .of a teacher •s instruction'el effectiveness. So«e of the 
items included in* such questionnaires' address only symptoms of 
instructional problems and. not underlying causes, a!id others ignc^re 
the fact that, learning and not teacher chatacteristics is the most 
important objective,. Similarly, almost anything that a faculty member 
does outside the classroom is now called faculty development, but in 
this melange, most development ptograms" have little potential for 
contributing significantly to instructional improvement. Two projects 
have bWen developed usi^ig a rigorous and systematic Approach to this 
pr5blem: the Center for Professional Dsvelopment with pilot programs 
on jgix campuses of the California State University and Colleges; and 
a comprehensive program at the UCLA Sct^ool of Dentistry. In the 
mtfltica«.pus project, a variety of efforts were undertaken on 
different campuses: narrowly-f ^^used faculty workshops, development 
of self -appraisal instruments, T:ourse^and curticular materials 
dfevelopment, institutional research projjects, and a program for - 
administrators only; It is felt th'a^^ttle change oOTur^ed or was 
initiated in this project. At UCLA,^n6wever, a nonf r^gmented, 
comprehensive faculty* development an^ instructional improvemer^t 
program^ addressing institutional' goals -is felt 'to be a success. The 
key is that the school as an organization, including bpth faculty a»nd 
administration, accepted respowsifaility for the consequences' of its 
educational programs. (MSE) ' 



Documents acquired by ERIC inclucfe many informal unpublished materials not av^able from other sources, EXJC makes every 
' effort to obtain the best copy available Nevertheless, items of marginal reproducibility, are often encountered and ttiis affects the 
quality of the rwicrofiche and hardcopy reproductions ERIC makes available via the ER^C Document Reproduction Seryice (EDRS). 
O is not responsible for the quality of the onginal document Reproductions suppbed by EDRS are the best that Can be made from 
ERLCWnal ^' • ' 



J u. « X - 




THE PATHWAYS ' kN\j /f^ITFALLS TO INSTRUCTIOI*AL IMPROVEMENT 



«*» lU * 



ERIC 



• by 
Clare Rose 

liiillSi Ey^aluatlon and Training Institute 

O*" ot »- ^ iT Z • , 



8 



^ WW t < z 2 
X 3 X K III ^ 



Student evaluations are neither reliable nor valid measures 'of' 
1^ , • a teacher^s instructional effectiveness, -and they .^cannot ^e used 
' f either to' evaluate faculty or to' improve instruction. Now that I 

have made that blasphemous statement,, let me clarify what I mean. 
^JStudent evaluatlori systems were developed iri the late ^1960s 
\JL# primarily to placatik angry students who demanded more direct in- 

paV Into the educationaj. system. In that sense, student ratings 
filled an important void in .higher ' education — ^hey provided 

the opportunity for students to express their opinion^ a^bout their 

/ 

teachers and their .courses and acknowledged tJje importance of. 

•client sati^i'ac^tion. Even more important, student evaluation 

systems brought renewed attention* to/ the impoVtance of teaching, 
* • 

' * acL unbelievably ^neglected aspect of higher education for far too 

/ 

^ . long. But, to suggest that stude'ht ratings are an effective aid 
fcrr improving ins^truction is akin to suggesting that fans' cheers 
. ^ and expletives can be used to improve 'the^quVlity of a baseball 
game. Only someone Gonvinaed that killing umpires was the key 
^ : .to winning games would make that assertion. 

- One- has only to look at the nt^ms- typical- of rating systems 
"v^ to understand the problem. , One /example , is the item ^^^Did the 

professor stimulate the students to high intellecttial effort?" 
How can being given a low rating on this item help you detect why 
yqu didn't and how^ you could do '^so In'the future? Conver's^ly, how 

*lnvited address presented at Southern Illinois University, 
Aprli, 1977 * ^ - ' " • ^ 



car¥%. high rating heli^ identify the magic formula so that it 

N ' ' ' ' 

can be 'ir^peated? - ' * 

\ - ' y " ' 

Another item oomfuonly us.ed on** student 'rating terms is the^ ' 
one that askfe ''Did the profe^por speak wit^ expresfel veness and 
variety of tone of voice?" * Again, whil,e a high rating might 
ejfctourage you to en?oll in Col\imbia Broadca&ting School, it' tells * 

.you nothing about whether the students learned -as a result of ' 

^ » * 

yaur mellifluous- tones. And isn't that what instructional im- 
provement is all about — increasing the probability that students 
ar^e learning as a result pf their spending 15-20 hours each week 
in class? ' ' ^ 

My second bit of heresy coTT^eTns Xfie contribirfc.ion of so- 
call'ed ''faculty development" programs to campus-wide instructional 
; * improvement. The melange oT activities that are oalled facu^lty 
development is' staggering — research projects, sabbaticals-, 
conferences.,' lectures, seminars, exchange progi^ams, growth con- 
tracts, instructional design projects, bag' lunches and workshops 
^ ' of every t^pe and variety. Almost anything a faculty member does 
these days ( outside of class , that is) is called faculty develop- 
r ment. 'But, regardless of the type oT activity, mq^t faculty de- 
velopment programs have little potential^f making a significant 
contribution to improving the quality of instruction in higher 
education. 

The reasons for the impotence of these programs are very 
simple. First, few faculty development ff'ograms mandate faculty 
participation. Although sanctioned by the administration;^ most 
programs malRtain a relatively low profile quite deliberately. 
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The assumption is that a strong, campus-wide appeal for faculty to 
further develbp thieir' professional- oompetencies such as improving 
their instruction might be viewed as-' a p6rsonal^ insult. Because 
few faculty are willing to acknowledge inadequacies in their in- 
structional competence, more seductive tac t ic s /tfmst be use'd to 
cajole, them^into some type of deve^lopment activity. Voluntary 
programs notoriously .attract thc^se faculty who are most ofSen to 
improvement - apd in thd major^y of cases/ that means the al- 
ready competent teacher. While' ^venk the best teachers can behefit 
from instructional improvement programs, the ones who really, need 
a rigorous program of devel6pment do not participate and yet' they 
are the ones who must do so if major improvements are. to be made 
in the quality of an inst i^^tujcion * s Instructional program. 

A s^ond fact^or that/ limits the potential of faculty develop- 
l 

ment programs to bring atiout significant inlstruct ional improvement 
is that few of these pro^rama*are designed systematically on the 
basis of the. goals of tilt college, -the departments and the related 
teaching needs, of the faJculty. Some programs exist in limbo -be- 
cause they were estab^is 
all, what college these 
of a faculty development 
so that -the college can 
Federal -monies so tempt 
velopment. In few such 



hed^ as tokens of a current fad. After 
days could" admit with grace to the lac\c 
program?' Other pi^ograms ^re established 



obtain a^share Of the foundatiori and 
ngly offered in the name of faculty de- 
cases fs the desire to i*ncrease faculty's 
teaching effectiveness 4 genuine concern* 

But, even in , tJ\e * case -^pf good intentions an^- the purest of 
motives , »many faculty development programs "are conceived in 
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isolation 'fi^'om the rest of %he institution.. Th^ as,sun>](^tibn Is" ^ 
that the mer^ existence of a P'aculty Program will somehow diff.use 
and affect all of the othe^ components in the institat ional 
system. These program pl-anners ignoi^e the ^fact that their college 
is a -complex organization composed of Int errelatetS and continuously 
interacting people, processes and purposes. If i\t is to be efa- 

\fective, a program for instructional improvemenfmust specially 
V^signed and integrated into the organisational system -and It ^ * 
mukt be compatible with the goals and^n.eeds ©f that $ystem. 

^Fina^lly, most faculty development* programs , particularity 
those t^hich are ^composed of a potpoUrri of workshops, focus on 

^process.— not outcomes. As such, t'h^'are Susceptible to an 

; early demise. Once 'the range/oT "saleable" workshop topics 
has been exhausted, the program fades into oblivion. * And oblivion 
may* well be beneficial, certainly as. far as instructional improve- 

. ment Is concerned. Devotees of "process" programs cleave 'stub- 
bornly to the importance of tWe classroom environment, the par- 
ticular Instructional methods used (lecture is a bad; dis- 

^ I. * » 

cussion is a goociX-and the way in which the faculty member per- 
i ^\ 

ceives ijis7her .rol^ as a teacher. No evidence has yet been found- 
to relate ^any of these variables to student learning.*' Research 
has. shown over and over again .that different teachers using dif- 
ferent instructional l^echniques in different class settings can be 

equally .effective in bi^inging about student learning with 'dif ^rent 

' ' ' ' 

types of students.' Sbm^ students learn better from the lecture 

method; 'Others require t;he. interact fon of a discussio.n mode. Some 

students respond to an authoritarian teaching style; others to a 



permissive one. "And'to confuse the issue eve-ji^urther , what . 
some^ students"' ^view as "exciting, otiiers vie^^ a-s dull! ' There is " 
"sim'pljir no single teacher' trait _ or ^ctTon th^t has proven to "be 
associated with student achievement. ., 
• . Does all of this mean that viable campus-wide instrui^tional 

^improvement is impossible? Not at all.^^ Wha1; it does mean is 

• t, ^ ■■ • ■ 

•that a -rigorous instructional improvement prog^am must be care*' 

^^^lly and syste matically planned. For optimum resultsj it' must ' 
be' designed as a campus-wide program based 'on a careful assessment 
of institutional and faculty needs. Jhe -link between faculty de- 
velopment and. instructio/al improvement cannot be assumed ipso 
.facto. Let me give you ivTo examples tp illustrate. 

The first example is the Center for Professional Development 
which was^ established in the Chanpellor's Office of the Cali- 
fornia State University and- Colleges , in Summer 197^^. The Center ' 
waf funded by^the Chancellor ' i Office and the Fund for the' I«m-' 
provemen,t of Postsecondary " Education , otherwise known as PIPSE 
or "tWq Fund", at approximately $.5 million over a three-year • 
per.lo^. The purpose of. the Center was to promote faculty develop-' 
merit/ in *.the hopes ofimproving instruction and the quality of 
edu/cation on the nineteen campuses in the systen^. Six campus'es' 
w^re selected to devel.op and test different kinds of pilot programs 
he six campus.esy ranged from a small, i^.elatively isolated college 
with 129 faculty W 3,500 students, tcf"^ large, metropolitan ' ~ ' 
university withv 9o\ faculty serving/o^t 26,000 students. It was- 
hoped' that definiti^^ guj.delines coi^oi^be developed for'-other 



Colleges and unl^ersi-fcies concerning which "^kinds of programs 
(or Rrogram elements) worked best in, whicTi^ways on -What kinds of 
campuses. , * 

■ ■ ' ' ■ - ^ \J 

'Two campus programs consisted primaro^ly of assorted work- 
Shops for small groups of faculty based on*a narrowly- fpcused 
need5 assessment of- consumers , that Is, a survey of , what, kinds 
of topics faculty were interested in hearing about. No assess-. 
*ment^ of. either institutional goals or student^ learning was under- 
taken, and, as a result, no strategy for> relating the program 
to instructional improvement could be developed. 

» 

Attendance at workshops during the first year was excellent 
on both campuses. Seventy-five percent -of the ^aculty^ on one 
cam^s and over 60^ on the other participated In a series of 
workshops on Piagetian theory, computer-assisted instruction, 
experiential learning, testing and grading and proposal writing. 
By the- end- of the first year, .there were no more "new" topics to 
motivate the-f acuity ^ s interest. Only_ a "hand^ful of faculty 
attended the remaining Sessions. ' 

*A third campus program consisted of the development of " 
diagnostic s^lf-appraisal instruments for faculty to use to / 
assess their, teaching capabiTities and effectiveness. Resource 
units that col^respbnded to the various dimensions of the instru- * 
ments were als© developed purportedly to help -faculty -imprdjve or . 
supplement the teaching skill deficiencies they had identified. 
The project staff 'compiled an^^ e^^tensive number of items and cate- 
gorized and coded them to form item pools from which self-appraisal 



forms could, be developed by individual faculty. -An equa,lly 
ext^ensive array of self-instructional materials were developed 
on topics such as the lecture method, disjcussion formats, grad- 
ing problems 5 etc. - . 

Again, hav^^ever, the needs assessment which formed th*^ 
base of the project, although conducted somewh'at more systema- 
tically, concentrated on the faculty, not on ins^t itut lonal needs 
or the problems associated ^with. student lea-^rning* . Even more' 
strange, despite the fact that in the needs assessment survey, 
the faculty overwhelmingly endorsed the idea and indicated their 
desire to ^use the'' self-evaluation instruments, very few were 
willing to do so once they were developed. Some departments 
jpefused to cooperate at all. It is highly doubtful that, this 
project will have any lasting impact on either the university or 
the handful of faculty who participated. 

A fourth campus focused on a mini-grant program for faculty 
to develop course and curricular materials. ^he problem with 

this program was simply that, like most mini-grant programs for 

- i\ 

faculty-initiated. projects , the products were of questionable / 

* . - 

instructional value. Most faculty are simply not trained to 
develop and' evaluate high quality instructional' materials . As 
a result ,^ while mini-grants, mignt serve as motivating forces for 
faculty to examine .their 'teaching , without ■ assistance from in- 

' * # 

structional design specialists, they rarely make any meaningful 
change in instructional practices. 

• An 'academically elite Institute was the f^al point for the 
faculty development program instituted at ano^Pt campus. The 



Institute was composed df and directed .by a group of 14 depart- 
mentally--nominated and presidentially-appointed fg^'cult^ who met 
regularly each week to rep6rt on carefully glanned institutional, 
research projects /elating" to many aspects of tea^ching and le'arn-^ 
ing on that campus. In addi t ion the Institute sponstrred one- 
day and; weekly seminar programs for the^ faculty at-large, infof- 
mal departmental and school - ''gatherings ^' and system-wide conf efefvces 

Although' this program also was not based pn a comprehensive 
institutional analysis, it was in large measure planned to deal 
with acknowledged concerns of student^^and faculty alike - par- 
ticularly that' the quality of education on the campus was suffe^ring' 
from a. pervasive lack of collegialit'y and sense of common purpos^.(^ 
The program was at least directed toward a defined institutional 
need rather than at a series of "interesting topics". 

An entirely different type of program was designed solely 
for administrators at ^the sixth campus and^ f or the most* part re- 
sulted from a conscient ibu^ institutional seTf-study and a 
university^^^wTJe commi1::rTrient to professional development and teach- . 
ifig improvement. This project consist,ed of a seminar program for 
department chairs and school deans w^hich focused on the professional 
responsibilities of administrator^ and their specific Vole and 
obj*ectives as managers and facilitators.' Approximately 95 percent 
of the deans and departmient chairs particlpaJ^^-if^- ttre^rogramT'^ ' 
and many inst itytidnal policies an^ practices affecting both student 
an^ faculty .have changed in a positive direction as a direct result 
of this program.' ' * • 



All of the camptis staffs ori^each of 'the six campiise's were, 
* » * ' ■ 

de'dicated and worked very hard tio make their projects successful. 
The. Center staff was equa'lly committed to the -program. Yet, i 
despite the ^st amounts of money, time and 'energy invested ?n . 
these projects, with the possible exception of the last two pro- 
grams (inscribed. It Is unlikely that- any broad based Institutional 
dhange , In either programs or policies, has taken or will ever 
take place . ^ • « ^ * • 

Now let me give you another' example — 'that of a systematically 
planned; comprehensive' ppgram of instructional Improvement and 
'faculty development. . This program was developed at the UCLA School 
of Dentistry 'but It is applicable to all ^^Institutions of higher 
education* Four primary assumptions provided the foundation 'for 
the program: 1) that the responsibility for faculty development 
and Instructional improvement rests with the institution or system 
a^ a whole; 2) that the goal of faculty developm.ent is the Im- ' 
provement of teaching and learning; 3) that .significant Improve- 
ment in the teaching-learning process mu^t be based upon rlgoEous, 
comprehensive Inst Itution-w^de Evaluation; ,and ^) that evaluation 
must focus on outcomes . — 'outcomes in terms of faculty motivation, . 
development and satl^sfactlon; the regj^onsiveness of course offer- 
-ings and currlcular sequencing to student s^needs and professional " 
Requirements; and most Importantly, outcoines in terms of student f 
learning and -development . ' ' ■ ' i 

Tl^ , first steps in the project were to sy>stematlcally develop 
measurable ^goals for the School to replace the jatalogue-t'ype broad 
mlssio^n statements that existed at the time, and to Involve, faculty 
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.-airectly^and deeply in the project from the beginning. First, 
.we began^workingiiwith a su6-comraittee of a'regular standing cur- ^ 
riculai* committee whicTi included^ one mem.lier. from ^ach of the three 
major ^curricular divisions in the scho9l, -a representative of the 
•student body ar^d' a member- of the administration. - Each^'membfer ^ ^ 'X! 
solicited general ideas from the faculty and students and drew up 
a tentative, list of issues^ directions and concerns which formed 
the basis for't;he school's first-ot'der goals. The goals were then 

• sent back to the entire faculty apd a 25 percent Sample of, students 
for review. Based upon their revisions- and suggestions, the goals 

; were refined, converted ir^to measurable objectives, resubmitted to. 

the faculty^ and after another round' of revision and review,' were 

f T ^ 

accepted. Similar' procedures' were then u^ed tcj establish goals 

for each of the ^14 sections within the school. 

* ( • . ' ' 

O^er 90 percent of the facul.ty and more than 60 percent of 
\ ' - ^ 

the students participated in developing 'the goals, and, as a result, - 

became increasingly enthusiastic ^ith the whole project. As ^the 

faculty clarified. more explicitly what, they^ wanted to teach in 

order to write their obectives, they began to question their 

■* » 

effectiveness and wanted to improve their current teacfilng skl^fts 
' as well as to add a broader range of teaching metH^|^ to tb^^ 
repetoire. ' * 

Als0;^s a result of definin^g section goals and developing ' . • 
Indices of their attainment, the faeulty^ came to realize that, 
student, complaints about their evaluation pt^ocedure^ Were justified. 
While few faculty quibblect wifeh the notion that, their main pur{Jlse* 
for being in the classroom was to bring about student learning, 

* > " \ • ' • ' . ■ ^ 
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When it came dowrif to examining their instruction for t^he purpose 

of improvement, n6t many of, then) used student* outcomes as the 

criterion of effectiveness. In manv ,.area§,^ .'fctculty ha^ no dej- 

finitive criteria for student pe.rf of mance-. At the seme time, 

the^' ex<)ected students to perform .4'up\ to 'S^tan&ciard" without. 

realizing t^hat they had not, estab^l-ished what't^t stSnda^d was. 

This awareness led^the faculty to „^sk for assistance in developing' 

appropriate tests and criteria for * evaluating t^j-eir Reaching ^ 

efifectivep-e^ss and their students^ performance. I'his'need in t\irK ' 
' • * * «» 

'led to a 'series c5f quarter long programs'* c?n evaluation, test 
construction, individualized instruc tj.on , student ;Learntng style's' 
^and multi-method approaches -to teaching. The faculty were so" 
enthusiastic that the administration designated one-h^lf ^ay each 
week. as , "Faculty Development .Day^. No courses were held 'and all 
laboratories were closed so that all faculty were free to attend 
the vac^ety of programs 'offered. -The^p'oint Is not that so many 
faculty participated' oj-^th^t they were so enthusiastia,' but- that 
the faculty development . and other- subsequent program's v/ere a direct 
result of institutional self-analysis and were therefore directly 
related^^t^/the needs Of the ^ institution an^ tshe needs of the' f^^lt 
witly respect to the institutional goals* „ , ' . 

'Another offshoot of the pro j ect ''was * that several faculty began 
working with us on developrpent of self-instruct ional ^modules . - . 

In fact, one entirq se'ction will be corfipletely 'modularized 'byj the . 
end of the next academic year. This, too, had an impact on the 
School ^nd fo^med^^he basis for* another series of faculty seminafs; 



•■"As 'word spread about the- achievement arid sVtisJtact i-on of. the- students 

. who used the newly developed materials, other faculty wanted to, 
develop 'modules for- the.ir courses • • ' . ' . ' 

' . In'order to ^ establis'h the continuing, cycle'of evaluation and 
.feedbaclc, required in the pr*ograrn plan, a survey of graduates \ 
was* conducted to det'erjnine their actual post-gradtation knowledge 
4nd behavior relevant -tp each of the goals which'^-had been estabir' 
Ushed.- The results of^the surv'ey w^^re- flsed tx)/ evaluate and 
modify 'the ' goals themselves and 'yearly graduate surveys provide 
•a Ipase'for on-going re vision of .goals , ,cu,rricula and ins true tllonal 

,> pi;ograms. The-.-cycie'- o/ change and l^enei^a^r will (Continue ar^ no- 
Houbt, additional programs needed by faculty and wiministrators' 
will be^ identified. * ' . ' ' , r 

An all-encompassing, institutional' proc.ess: of improvement • 
h?is been gerierated^that includes elements of what some people 

.'differentiate a^ faculty development, instructional improvement, 
carricular development and organizational d'evelopment . -We see 

v^hese elements as integrally connected parts, of a total improve- v 
ment program. -All aspects of 'the process are evaluated in terms', 

* T 't 4 ^ 

^ of their contributiCn- to students' personal development and 

acadefnic achievements I . Becau.se this^ process has been institu- , 
''tionalizad throughout the scho(^ imjirbvem^nt • in t^e quality of ^ 
, the educati6n provided has b'een significant, if not mpnuftientai. 
.The "key. is that'1;he school as . a'n organization accepted re- 
s'ponsibility for the consequences 'Of its educational programs. 
• Faculty and chairmen accepted responsibility for the results of . 



their dei5artmental programs ahd finally, faculty as indivldoials * 
accepted re^onsibility for the 'corfsequences of their instruction. 
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Collective responsibility for the consequences of instructional 
^ programs and Evaluation 'in terms of student outcomes are the . ' 
^two -fessential ingredients for^ tuildifig the_ road to instructional 
improvement . ' . ' . - * ' 
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